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ABSTRACT 


Metacognition is an aspect in self-regulated learning and is necessary to achieve such learning in an effective and 
efficient manner. However, it is not always easy and accurate for learners to monitor or assess their own metacognition. 
In this study, we designed a learning analytics dashboard to improve self-regulated learning in online environments 
through the collection and analysis of learning log data. There are two separate dashboards used in our system: 
a knowledge monitoring dashboard and a strategy use dashboard. The knowledge monitoring dashboard is designed to 
support the knowledge monitoring skills of learners, allowing them to monitor their prior knowledge, whereas the 
strategy use dashboard is designed to help learners develop metacognitive skills of planning, monitoring, and regulation. 
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1. INTRODUCTION 


Self-regulated learning (SRL) is considered an effective way to achieve academic goals such as test grades 
and academic skills in school, or even the capacity to self-instruct in the future (Zimmerman, 2002). SRL is 
considered to be a self-directive process, through which SRL learners transform their mental abilities or 
intelligences into academic skills (Zimmerman, 1998). SRL is closely linked to the concept of self-generated 
thoughts, feelings, and behaviors, particular in terms of metacognition and motivation (Tobias and Everson, 
2009; Zimmerman, 1998; Zimmerman, 2002). Considering the metacognition in SRL, it is necessary to allow 
learners to make sense of their own learning process and take responsibility for their own learning. 

As information and communication technologies advance, various types of data can be collected, not only 
test scores and psychometric data, but also learning logs to allow learning behaviors to be analyzed through a 
learning analytics (LA) approach. This new approach provides a new direction for recording and monitoring 
learning and for supporting learners in developing their metacognitive skills through SRL. 

This paper aims to design a learning analytics dashboard to support the metacognition skills of learners by 
visualizing their log data during the SRL process. 


2. THEORETICAL BACKGROUND 


2.1 Metacognition and SRL 


SRL strategies are defined as “actions and processes directed at acquiring information or skill that involve 
agency, purpose, and instrumentality perceptions by learners” (Zimmerman, 1989). Self-regulation is an 
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important construct of learning that includes both metacognition and motivation dimensions (Tobias and 
Everson, 2009). 

Metacognition, which refers to the ability to set goals or plans, monitor and evaluate one’s own learning 
(Tobias and Everson, 2002), is one aspect of SRL and is necessary in regulating learning in an effective and 
efficient manner (Tobias and Everson, 2009). Many models of metacognitive strategies concerning SRL 
include three general scales, namely, planning, monitoring, and regulating (Hofer et al., 1998; Pintrich et al., 
1993; Pintrich, 1999). 

Planning activities contain the usage plan of several cognitive strategies, such as setting goals and criteria 
or standards for evaluation, clarifying the aim of the learning activities, and conducting task analyses of 
problems (Belfiore and Hornyak, 1998; Hofer et al., 1998; Zimmerman, 1989). By planning activities, 
learners are expected to develop a cognitive strategy utilization, activate their relevant prior knowledge and 
experience, and improve their organization and understanding of the learning content (Hofer et al., 1998). 

Monitoring activities include tracking one’s attention during the learning processes, checking the degree 
of understanding, and use of learning or examination strategies (Hofer et al., 1998; Lan, 1998) by observing 
or recording one’s own learning behaviors or performance (Belfiore and Hornyak, 1998; Zimmerman, 1989). 
Monitoring one’s own learning behaviors and processes is an important aspect in SRL because it helps 
learners to be aware of their weaknesses or deficiencies in their learning processes and regulates their 
learning strategies or academic behaviors during the next phase (Hofer et al., 1998; Winne and Stockley, 
1998). 

Regulation strategies are related to and based upon monitoring strategies, and they aim at improving the 
learning behaviors of learners and repairing deficits in their comprehension (Hofer et al., 1998). For example, 
learners focus on one part of a text, summarize what they have read, and reread the texts based on their 
comprehension, adjusting their learning/reading pace according to their learning ability, or choosing 
course-related materials (Hofer et al., 1998; Thiede et al., 2009). 

However, it has been shown that metacognitive skills are required to monitor one’s own prior knowledge 
for effective and efficient learning (Tobias and Everson, 2002, 2009). The lack of an ability to distinguish 
correctly what learners know or have learned, from what they do not know or need to learn, will lead to a 
failure to engage in more advanced metacognition strategies (Tobias and Everson, 2009; Yen et al., 2018). 
This is because metacognition strategies are highly ordered processes used to monitor and coordinate basic or 
complex cognitive processes. However, learners with low or underdeveloped knowledge monitoring skills 
are considered to be inefficient at reading the learning materials, leading to a failure to apply a reasonable 
time allocation and sequence management. Thus, Tobias and Everson (2009) emphasized the importance of 
knowledge monitoring (KM), which is the metacognitive ability to monitor one’s prior knowledge, and 
placed monitoring knowledge at the bottom of all metacognitive phases, indicating that it is the fundamental 
component of metacognition. 

Therefore, in our study, a dashboard for supporting the metacognition of learners was designed based on 
the dimensions of knowledge monitoring, planning, monitoring, and regulation. 


2.2 Metacognition and Learning Analytics 


Although metacognition is an important and effective way for learners to monitor and regulate other 
cognitive strategies, metacognition measurements are facing considerable challenges. When observing the 
performance of learners, or assessing their metacognition through self-reported inventories, the validity of the 
inferences of these evidence-based approaches, as well as the accuracy and candor of a self-assessment, are 
problems and issues to be overcome (Tobias and Everson, 2009; Winne and Nesbit, 2009). For example, 
learners cannot observe or remember all events or their learning behaviors during their learning activities. In 
addition, their perception will be influenced by their previous experience, the pace of events, biases, their 
memory, and other factors. 

Concerning these issues in measuring metacognition, information and communication technology (ICT) 
has shown significant potential in collecting learning log data. As ICT advances, the types of log data and the 
methods of data collection vary. For example, the interaction between or across learners, or between learners 
and computers, can be collected to show the software or system features the learner uses (Tobias and Everson, 
2009). In addition, the system operations during the entire learning process can also be collected, such as 
copying or highlighting texts, or adding annotations, to present the learning behaviors of learners (Tobias and 
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Everson, 2009; Yamada et al., 2017; Yen et al., 2018). 

In Yamada et al.’s (2017) study, the correlations between SRL awareness, learning performance, and 
behaviors, such as the duration of the slide pages read, the adding of markers, and annotations, were analyzed. 
As the results indicate, the slide pages that the learners read for a certain duration (from 240 to 299 s) have 
positive effect on their annotation utilization and learning performance improvement. 


2.3 Visualization 


Yen et al. (2018b) pointed out that visualization is useful for learners to be aware of what they have been 
doing and what they should do by making such information salient for them. For example, in online inquiry 
activities, the visualization of driving questions, a set of goals, or the structure and sequence and associations 
of tasks, could help learners conduct such activities more smoothly, such as planning, monitoring, and 
evaluating during SRL (Manlove et al., 2006; Zhang and Quintana, 2012). 

However, such visualization tools have been used in domain-specific systems. Considering the utilization 
of visualization to support the general skills and metacognition of learners, visualization tools should be 
applied in domain-general systems. 

Learning analytics has many advantages including the continuous and automatic collection of the 
learners’ data in ubiquitous learning environments, which are not interfered with by the learners’ engagement 
or intervention of external instructors (Yamada et al., 2016; Yin et al, 2015). Teachers can understand 
learner’s learning behaviors out of class and informal learning settings (Shimada et al, 2015). Integrating 
these advantages into a dashboard, learning analytics can be provided through a visual display of the data 
information, informing the learners regarding undetectable patterns of their progress and behaviors (Teasley, 
2017) by collecting and analyzing the data generated from online learning platforms, primarily a learning 
management system (LMS). As Verbert et al. (2014) reviewed in their research, various types of data can be 
collected and visualized which are relevant with learners’ behaviors, including learners’ social interaction 
with other learners or systems, resource use which is used in examples of recommender systems, to estimate 
learners’ knowledge levels, time spent which is visualized for instructors and learners to help them 
understand learners’ time and efforts spent on specific learning sessions, and also the test and self-assessment 
results that capture knowledge and self-awareness levels. 

Kim et al. (2016) examined the effects of a learning analytics dashboard, which presents students with 
frequency data of certain activities in a college online course. As the results indicate, students who receive a 
dashboard treatment achieve a higher final score than those who do not; however, the dashboard usage 
frequency does not have a significant impact on the learning achievement. 

Roberts et al. (2017) emphasized student involvement in the dashboard development process and 
analyzed the students’ attitudes toward learning analytic dashboards. As their results indicate, students want 
access to additional resources easily or general links to an assessment or unit, numerical (class 
mean/median/max/min) or graphical statistics, and an anonymous comparison with their peers, among other 
information. 

Based on a review of previous studies, and taking the good features pointed out above into account, we 
designed a dashboard to support learners’ KM, learning process monitoring, and regulation during SRL. 


3. DESIGN OF LA DASHBOARD FOR METACOGNITION 


Our learning-analytics based dashboard consists of two parts: knowledge monitoring and use strategies. In 
the use strategy phase, three dimensions, namely, planning, monitoring, and regulating, are included. 

The learning analytics dashboard was designed based on the BookRoll system (see Figure 1), and the data 
collection is conducted using this system. BookRoll is an e-book system applied to distribute digital versions 
of course materials. Students can highlight the contents or add annotations within the digital materials. 
Furthermore, the accumulated data on the students’ use of the BookRoll system functions can be collected 
and used to visually represent their processes in viewing the materials (Ogata et al., 2017). 
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Figure 1. Interface of BookRoll system 


3.1 Knowledge Monitoring 


Based on the “hierarchy of metacognitive processes” proposed by Tobias and Everson (2009), in which 
knowledge monitoring is placed in the fundamental phase of the metacognitive processes, one component of 
our dashboard is to support the knowledge monitoring of learners. 

With metacognitive skills used to monitor one’s own prior knowledge, learners can adopt more 
appropriate learning strategies. For example, they can decide to skip parts of the learning materials that they 
are familiar with or have already mastered or focus on the parts they do not yet understand and seek help 
from others based on their recognition that their knowledge needs to be improved or supplemented (Tobias 
and Everson, 2009). Thus, in our study, we designed a dashboard to support learners in knowledge 
acquisition by providing them access to recommended learning materials related to the knowledge they do 
not know or are not familiar with. 

To assess the monitoring of prior knowledge of the learners, Tobias and Everson (2002) developed a 
knowledge monitoring assessment (KMA). With KMA, learners are required to estimate whether they can 
solve each problem, and then take a test on the same content; the accuracy of KM of the learners can be 
inferred through a comparison of the accuracy between the metacognitive estimation and their test 
performance. 

In addition, using a KMA, the knowledge monitoring of the learners can be divided into four types: 
(1) estimated as known and with a correct answer on the test (+ +), (2) estimated as known but with an 
incorrect answer on the test (+ -), (3) estimated as unknown but with a correct answer (- +), and (4) estimated 
as unknown and with an incorrect answer (- -) (Tobias and Everson, 2002). 

The first (+ +) and fourth (- -) types of learners show consistency between their KM and the actual 
knowledge acquisition, whereas there is a difference in that the fourth type needs greater effort in the learning 
content and knowledge. By contrast, the second (+ -) and third (- +) types of learners show a discrepancy in 
estimating and utilizing their knowledge and indicate an insufficient KM skill of metacognition. On our 
designed dashboard, the estimation and actual results of the tests, as well as the pattern of the KM, are 
presented to learners, who are divided into one of four KM categories according to the most frequent pattern 
they demonstrate. All of these categories are presented to the learners through a graph, with the test scores 
shown along the x-axis, and the numbers of their “correct” and “incorrect” judgements shown along the 
y-axis. 

In Figure 2, the design of the “knowledge monitoring dashboard” is shown. On this dashboard, the results 
of the tests and their knowledge estimation, as well as the KM pattern according to such data, are provided to 
learners. For learners who have difficulties in knowledge acquisition, the link between the contents they do 
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not understand with the related digital learning materials will also be provided. When learners make a 
mistake on the test, the “link to the related materials” icon is presented next to the answer. In addition, when 
they click the icon, learners can access the recommended materials on the BookRoll system through the link. 
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Figure 2. Design of knowledge monitoring dashboard 


3.2 Planning 


As we reviewed in the theoretical background section, the aim of planning is the development of cognitive 
strategies through activities such as organizing the contents and activating prior knowledge (Hofer et al., 
1998). In our study, we considered designing our dashboard in a way that does not support the planning 
process, but applies a preplanning phase to help learners organize their learning content and compared it with 
that of other learners. 

Thiede and Anderson (2003) proposed that, after reading a text, it is effective to improve the learner’s 
understanding and comprehension by summarizing the contents using keywords and then rereading the text 
that the learner is unfamiliar with. Considering the effectiveness of a summarization approach, we designed 
the dashboard to support the metacognition of knowledge organization by showing learners how to utilize 
BR-Map. 

BR-Map is a concept map tool using logs stored on the BookRoll system (see Figure 3). BR-Map can 
read and list all logs of highlights and annotations from the BookRoll database (the operations applied when 
learners read e-learning materials) and display them as objects on the left-side pane. Learners click an object 
on the left-side pane, and drag-and-drop it into the right-side pane, namely, the “concept map area,” and 
create concept map(s) by connecting the objects using an “arrow” (Yamada et al., 2018). 
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Figure 3. Interface of BR-Map 
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With our dashboard, to improve the learner’s perception of the knowledge organization and construction, 
we designed a function for presenting the slide pages applied when making the user’s own BR-Map, or for 
adding highlights or annotations to these pages; the details of BR-Map of the learner or other students are 
presented when clicking on the slide pages icon (Figure 4). 


3.3 Monitoring and Regulation 


Monitoring strategies alert learners to concentrate on and understand their learning processes to regulate 
learning. During the monitoring phase, learners are expected to focus on the processes rather than the 
outcome only (Zimmerman, 1998). Many activities can be conducted to support self-monitoring, such as 
tracking the learner’s attention to the learning materials, determining how well the learner reads a text, or 
checking for the learner’s understanding of a particular lecture. 

Therefore, in our design, how learners read a learning text, and the time they spend on each page, is 
shown on the dashboard (in Figure 4). A thicker arrow means more learners are reading on the same path, 
and the intensity of the color indicates the time they spend on each page; the darker the color, the more time 
the learners required. 

Because the point of a self-evaluation is a comparison of one’s own performance or behaviors based on 
certain criteria or standards (Belfiore and Hornyak, 1998), it is necessary to provide learners with the criteria 
or standards to make a judgement regarding whether they finished the work well. However, it is difficult to 
generate standard learning behaviors, and considering the class average information and an anonymous 
comparison to the learner’s peers, which is information sought out by students, we provide both the 
individual utilization of BookRoll system tools, and a class average utilization. Learners can switch the frame 
by clicking the “Me” and “Class” icons. 

In addition, when learners click the slide page, the detailed information will be provided on the dashboard, 
including the numbers and contents of the markers (highlighting), the numbers of annotations, and the time 
they spent on each page (Figure 5). By checking the information of the learning materials and comparing the 
information with other students, learners are expected to become aware of their insufficient understanding of 
the materials, as well as the contents they should concentrate on, and thus finally improve their selection of 
cognitive strategies and learning behaviors. 
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Figure 4. Design of strategy using dashboard | - slide reading path and time 
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Figure 5. Design of strategy using dashboard 2 - analysis tools 


4. CONCLUSION AND FUTURE STUDIES 


In the present study, we designed a learning analytics dashboard to support the metacognition of learners 
regarding the dimensions of knowledge, planning, monitoring, and regulation. However, we have yet to 
develop this dashboard fully, and we were unable to clarify whether it achieves the expected level of 
effectiveness. Therefore, in a future study, we will develop this dashboard further and evaluate the dashboard 
especially in the dimension of effectiveness, efficiency, usefulness, and usability (Verbert et al. 2014) 
through a practical experiment. 
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